Preserve file encoding in file-based app source files by Copilot · Pull Request #52055 · dotnet/sdk

Copilot · 2025-12-06T20:00:04Z

Original prompt

This section details on the original issue you should resolve

<issue_title>file-based apps: dotnet add file.cs package changes encoding of file.cs</issue_title>
<issue_description>### Describe the bug
The SDK changes encoding of file-based app files when adding packages.

To Reproduce

1 - Create a a file.cs (UTF-8 encoded)
2 - Add a shebang : #!/usr/bin/env dotnet run and chmod +x file.cs
3- Run it: ./file.cs
4 - Add any package dotnet add file.cs AnyNugetPackageIdHere => file.cs is converted to UTF-8 with BOM
5 - Running ./file.cs again after the package is added results in failure:
> ./file.cs
./file.cs: line 1: #!/usr/bin/env: No such file or directory
Further technical details

.NET 10 SDK, 1.0.100</issue_description>

<agent_instructions>Look at the root cause analysis and suggested fix in the issue comments</agent_instructions>

Comments on the Issue (you are @copilot in this section)
@DamianEdwards Analysis from Copilot:
Root Cause Analysis

The issue is in the SourceFile struct in [src/Cli/Microsoft.DotNet.FileBasedPrograms/FileLevelDirectiveHelpers. cs](https://github.com/dotnet/sdk/blob/e300dea82e0248b0801f706914c96073c1bd5350/src/Cli/Microsoft.DotNet. FileBasedPrograms/FileLevelDirectiveHelpers.cs#L256-L274):
public void Save()
{
    using var stream = File.Open(Path, FileMode.Create, FileAccess. Write);
    using var writer = new StreamWriter(stream, Encoding.UTF8);
    Text.Write(writer);
}
Problem: Encoding. UTF8 (the static property) emits a UTF-8 BOM by default. When dotnet add file. cs package <package> modifies the file and saves it, the BOM (0xEF 0xBB 0xBF) is prepended to the file content.

On Unix-like systems, the shebang #!/usr/bin/env dotnet run becomes <BOM>#!/usr/bin/env dotnet run, which the kernel doesn't recognize as a valid interpreter directive, causing:
./file.cs: line 1: #!/usr/bin/env: No such file or directory
Fix: The Save() method should preserve the original file's encoding/BOM characteristics. Specifically, for files without a BOM, it should use:
new UTF8Encoding(encoderShouldEmitUTF8Identifier: false)
The Load() method should also detect and store whether the original file had a BOM so it can be preserved on save. </comment_new>

Fixes file-based apps: dotnet add file.cs package <package> changes encoding of file.cs #52054

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

- Modified SourceFile.Load() to detect UTF-8 BOM in files - Modified SourceFile.Save() to preserve original BOM state - Added HasUtf8Bom property to SourceFile struct - Updated InternalAPI.Unshipped.txt with new API members - Added tests to verify BOM preservation (no-BOM and with-BOM cases) Fixes issue where dotnet add package command was adding BOM to files that originally didn't have one, breaking shebang scripts. Co-authored-by: DamianEdwards <249088+DamianEdwards@users.noreply.github.com>

src/Cli/Microsoft.DotNet.FileBasedPrograms/FileLevelDirectiveHelpers.cs

jjonescz

@copilot

src/Cli/Microsoft.DotNet.FileBasedPrograms/FileLevelDirectiveHelpers.cs

test/dotnet.Tests/CommandTests/Run/FileBasedAppSourceEditorTests.cs

- Use SourceText's Encoding property instead of custom HasUtf8Bom tracking - Use stackalloc for 3-byte buffer on .NET Core (with fallback for netstandard2.0) - Update tests to use _testAssetsManager.CreateTestDirectory() infrastructure - Add issue reference (#52054) to test docs - Remove unnecessary API surface from InternalAPI.Unshipped.txt Co-authored-by: jjonescz <3669664+jjonescz@users.noreply.github.com>

jjonescz

@copilot

test/dotnet.Tests/CommandTests/Run/FileBasedAppSourceEditorTests.cs

src/Cli/Microsoft.DotNet.FileBasedPrograms/FileLevelDirectiveHelpers.cs

- Remove custom DetectEncoding() method and use SourceText.From(stream, encoding: null) - SourceText automatically detects encoding including BOM when encoding is null - Move <see href/> tags inside <summary> elements for better documentation - Simplifies implementation while maintaining full functionality Co-authored-by: jjonescz <3669664+jjonescz@users.noreply.github.com>

Copilot

Pull request overview

This PR fixes a critical bug where dotnet add file.cs package <package> was inadvertently adding a UTF-8 BOM to file-based app source files, breaking shebang (#!) scripts on Unix-like systems. The fix preserves the original encoding state (with or without BOM) by leveraging Roslyn's SourceText encoding detection and preservation capabilities.

Key Changes:

Modified SourceFile.Load() to auto-detect encoding by passing encoding: null to SourceText.From()
Updated SourceFile.Save() to use the detected encoding from SourceText.Encoding property
Added comprehensive tests verifying BOM preservation for both cases (with and without BOM)

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File	Description
src/Cli/Microsoft.DotNet.FileBasedPrograms/FileLevelDirectiveHelpers.cs	Updated `SourceFile.Load()` to auto-detect encoding and `SourceFile.Save()` to preserve it using `SourceText.Encoding`
test/dotnet.Tests/CommandTests/Run/FileBasedAppSourceEditorTests.cs	Added two test cases: `PreservesNoBomEncoding()` for files without BOM and `PreservesBomEncoding()` for files with BOM

- Add PreservesNonUtf8Encoding test to verify UTF-16 encoding is preserved - Tests that files with UTF-16 LE encoding (with BOM 0xFF 0xFE) maintain their encoding - Ensures SourceText's auto-detection works for encodings beyond UTF-8 - All encoding preservation tests now pass (UTF-8 no BOM, UTF-8 with BOM, UTF-16) Co-authored-by: jjonescz <3669664+jjonescz@users.noreply.github.com>

333fred

A few minor comments on the tests.

test/dotnet.Tests/CommandTests/Run/FileBasedAppSourceEditorTests.cs

Co-authored-by: Fred Silberberg <frsilb@microsoft.com>

jjonescz · 2025-12-12T10:59:14Z

@333fred @RikkiGibson for another look, thanks

test/dotnet.Tests/CommandTests/Run/FileBasedAppSourceEditorTests.cs

RikkiGibson

LGTM, comment is not blocking.

- Replace Assert.StartsWith with Assert.Equal to verify the entire file content - Ensures the test validates that Save() correctly writes the complete content - Makes the test more robust by checking the full output including the added package directive Co-authored-by: jjonescz <3669664+jjonescz@users.noreply.github.com>

test/dotnet.Tests/CommandTests/Run/FileBasedAppSourceEditorTests.cs

- Replace raw string literal with explicit \n characters in expected content - Raw string literals with indentation were causing blank lines to be stripped - Explicit newlines ensure consistent comparison across platforms - Fixes test failure on Windows where line endings differ Co-authored-by: jjonescz <3669664+jjonescz@users.noreply.github.com>

jjonescz · 2025-12-15T15:34:25Z

/backport to release/10.0.2xx

github-actions · 2025-12-15T15:34:38Z

Started backporting to release/10.0.2xx (link to workflow run)

Initial plan

92f12aa

Copilot AI assigned Copilot and DamianEdwards Dec 6, 2025

Copilot started work on behalf of DamianEdwards December 6, 2025 20:00 View session

Copilot AI changed the title ~~[WIP] Fix SDK encoding issue when adding package to file-based apps~~ Preserve UTF-8 BOM state in file-based app source files Dec 6, 2025

Copilot AI requested a review from DamianEdwards December 6, 2025 20:19

Copilot finished work on behalf of DamianEdwards December 6, 2025 20:19

DamianEdwards requested a review from jjonescz December 6, 2025 20:49

DamianEdwards assigned jjonescz Dec 6, 2025

DamianEdwards reviewed Dec 6, 2025

View reviewed changes

src/Cli/Microsoft.DotNet.FileBasedPrograms/FileLevelDirectiveHelpers.cs Outdated Show resolved Hide resolved

This was referenced Dec 6, 2025

SSL Template Issue Causes Missing OSX Templates #51198

Open

Could Not Resolve SDK - repo testfx issue #51530

Open

jjonescz reviewed Dec 7, 2025

View reviewed changes

jjonescz added the Area-run-file Items related to the "dotnet run <file>" effort label Dec 7, 2025

Copilot started work on behalf of jjonescz December 7, 2025 20:45 View session

Copilot AI requested a review from jjonescz December 7, 2025 20:58

Copilot finished work on behalf of jjonescz December 7, 2025 20:58

jjonescz reviewed Dec 7, 2025

View reviewed changes

test/dotnet.Tests/CommandTests/Run/FileBasedAppSourceEditorTests.cs Show resolved Hide resolved

src/Cli/Microsoft.DotNet.FileBasedPrograms/FileLevelDirectiveHelpers.cs Outdated Show resolved Hide resolved

Copilot started work on behalf of jjonescz December 7, 2025 21:16 View session

Copilot AI requested a review from jjonescz December 7, 2025 21:27

Copilot finished work on behalf of jjonescz December 7, 2025 21:27

jjonescz approved these changes Dec 8, 2025

View reviewed changes

jjonescz requested a review from a team December 8, 2025 09:11

jjonescz marked this pull request as ready for review December 8, 2025 09:12

Copilot AI review requested due to automatic review settings December 8, 2025 09:12

Copilot started reviewing on behalf of jjonescz December 8, 2025 09:12 View session

Copilot AI reviewed Dec 8, 2025

View reviewed changes

Copilot started work on behalf of jjonescz December 9, 2025 11:46 View session

Copilot AI changed the title ~~Preserve UTF-8 BOM state in file-based app source files~~ Preserve file encoding in file-based app source files Dec 9, 2025

Copilot AI requested a review from jjonescz December 9, 2025 11:55

Copilot finished work on behalf of jjonescz December 9, 2025 11:55

MiYanni approved these changes Dec 11, 2025

View reviewed changes

333fred reviewed Dec 11, 2025

View reviewed changes

Use collection expressions in asserts

07074b6

Co-authored-by: Fred Silberberg <frsilb@microsoft.com>

jjonescz requested review from 333fred and RikkiGibson December 11, 2025 10:33

build-analysis bot mentioned this pull request Dec 11, 2025

[failing test] Microsoft.NET.Build.Tests.GivenThatWeWantToBuildAWindowsRuntimeComponent.ManagedWinRTComponentCanBeReferenced #52131

Closed

333fred approved these changes Dec 12, 2025

View reviewed changes

RikkiGibson reviewed Dec 13, 2025

View reviewed changes

test/dotnet.Tests/CommandTests/Run/FileBasedAppSourceEditorTests.cs Outdated Show resolved Hide resolved

RikkiGibson approved these changes Dec 13, 2025

View reviewed changes

Copilot started work on behalf of jjonescz December 15, 2025 11:25 View session

Copilot finished work on behalf of jjonescz December 15, 2025 11:35

jjonescz approved these changes Dec 15, 2025

View reviewed changes

jjonescz reviewed Dec 15, 2025

View reviewed changes

test/dotnet.Tests/CommandTests/Run/FileBasedAppSourceEditorTests.cs Outdated Show resolved Hide resolved

Copilot started work on behalf of jjonescz December 15, 2025 13:55 View session

Copilot AI requested a review from jjonescz December 15, 2025 14:07

Copilot finished work on behalf of jjonescz December 15, 2025 14:07

jjonescz merged commit b300744 into main Dec 15, 2025
29 checks passed

jjonescz deleted the copilot/fix-file-encoding-bug branch December 15, 2025 15:33

github-actions bot mentioned this pull request Dec 15, 2025

[release/10.0.2xx] Preserve file encoding in file-based app source files #52159

Merged

dotnet-maestro bot mentioned this pull request Dec 16, 2025

[main] Source code updates from dotnet/sdk dotnet/dotnet#3754

Merged

Conversation

Copilot AI commented Dec 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

To Reproduce

Further technical details

Comments on the Issue (you are @copilot in this section)

Root Cause Analysis

Uh oh!

Uh oh!

jjonescz left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jjonescz left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

333fred left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jjonescz commented Dec 12, 2025

Uh oh!

Uh oh!

RikkiGibson left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

jjonescz commented Dec 15, 2025

Uh oh!

github-actions bot commented Dec 15, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Copilot AI commented Dec 6, 2025 •

edited

Loading